Subject image extraction device

ABSTRACT

A main image pickup section for picking up a main image and two or more subsidiary image pickup sections for picking up subsidiary images are used to shoot a subject. A 3-D object on which a predetermined pattern is drawn is also shot by the same image pickup sections, and parameters representing relative positions and attitudes of the image pickup sections are obtained using the images of the 3-D object. Feature points are extracted from each of the main image and the subsidiary images, and 3-D coordinates corresponding to each feature point in the main image are determined using the parameters and the feature points in the subsidiary images. Subsequently, judgment is executed and each of the feature points in the main image is accepted if the 3-D coordinates is within a designated 3-D space domain. Subsequently, subject outline points are extracted from the accepted feature points. Then, part of the main image which is surrounded by the subject outline points is extracted as the subject image. According to the device, subject image extraction can be extracted with high accuracy, without information of the subject or the image pickup device, without control or measurement of the positions of the image pickup devices, and with easy setting of a threshold value.

BACKGROUND OF THE INVENTION

The present invention relates to a subject image extraction device forextracting an image of a desired subject from an inputted image, and inparticular, to a subject image extraction device which can extract animage of a subject which existed in a designated 3-D space domain at themoment of shooting.

DESCRIPTION OF THE PRIOR ART

Various kinds of methods are proposed for extracting an image of adesired subject or object from an inputted image, such as a methodutilizing prestored information concerning the subject as disclosed inJapanese Patent Application Laid-Open No.HEI7-220095 (hereafter,referred to as ‘document No.1’), a method utilizing two or more imagepickup devices (cameras) as disclosed in Japanese Patent ApplicationLaid-Open No.HEI7-177423 (hereafter, referred to as ‘document No.2’) andJapanese Patent Application No.HEI8-271914 (hereafter, referred to as‘document No.3’), etc.

In the method utilizing prestored subject information which is disclosedin the document No.1, information concerning the image pickup devicesuch as the focal length of the image pickup device, the size of a pixelof a sensor of the image pickup device, the number of pixels of thesensor, etc. and information concerning the subject such as the size ofthe subject, the distances between the center point on the subject andparts of the subject, etc. are preliminarily recorded and stored in astorage unit. Then, the size of the subject in the image which has beenpicked up by the image pickup device is estimated using the informationconcerning the image pickup device and the subject. Meanwhile, aplurality of parts of the image which are considered to be candidates ofthe image of the subject are extracted from the image. Then, it isjudged whether each of the candidate parts is the genuine subject imageor not, by comparing the size of the candidate part with the estimatedsize of the subject in the image. And one candidate part whose size isthe nearest to the estimated size of the subject in the image isextracted as the image of the subject.

In the method utilizing two or more image pickup devices which isdisclosed in the document No.2, corresponding points in each image whichhave been picked up by the two or more image pickup devices are searchedfor, and disparities between the images with regard to the correspondingpoints are obtained using the difference of coordinates between thecorresponding points in each image. Then, part of an image whosedisparity between the images are large (i.e. parts which existed in theforeground near to the image pickup devices at the moment of shooting)are extracted as the image of the subject.

In the method utilizing two or more image pickup devices which isdisclosed in the document No.3. The subject is shot in the centers ofimages by the two or more image pickup devices, and the aforementionedsearch of point correspondence (hereafter, referred to as ‘pointcorrespondence search’) is executed from the center of the images. Then,outline point judgment (i.e. judgment whether a point in an image is apoint on the outline of the subject or not) is executed using 3-Dcoordinates (corresponding to the point) which are calculated using thecorresponding points obtained by the point correspondence search, andthe outline points are used for extraction of the subject image. Evenwhen false correspondence occurred in the point correspondence searchand parts of the outline points dropped out from the extraction, suchparts are restored by means of outline restoration assuming continuity.

However, in the method disclosed in the document No.1, theaforementioned information concerning the subject such as the size ofthe subject, the distances between the center point on the subject andparts of the subject, etc. and the information concerning the imagepickup device such as the focal length of the image pickup device, thesize of a pixel, the number of pixels, the distance between the subjectand the image pickup device, etc. are needed to be prepared, and thusimages of subjects whose information have not been prepared areimpossible to be extracted from the image which has been picked up bythe image pickup device. Further, according to the method, either thedistance between the center point on the subject and part of the subjector the distance between the principle point of the image and part of theimage is necessary as the information. In the case where the formerdistance is used, expensive devices such as an infrared irradiation unitetc. are necessary for obtaining the distance. And in the case where thelatter distance is used, methods such as autofocusing, which does notoperate stably under insufficient shooting conditions such as poorlighting etc., have to be employed, and thus stable extraction of thesubject image is difficult.

In the method utilizing two or more image pickup devices and judging thedepth using the disparity, the accuracy of subject image extraction ishighly dependent on setting of a threshold value (in disparity) which isused to distinguish between the background and the foreground, and thesetting of the disparity threshold value for accurate subject imageextraction is very difficult for the operator of the subject imageextraction device since the disparity is not a value which can bedirectly measured. The method of the document No.2 tries to resolve theproblem by obtaining 3-D coordinates using the disparities. However, themethod needs troublesome control or measurement of relative positions ofthe two or more image pickup devices, angles between the optical axes ofthe image pickup devices, etc. Further, according to the method, whenfalse correspondence occurred in the point correspondence search, thepart where the false correspondence occurred is extracted as noise, orin other words, parts of the subject image drop out.

In the method disclosed in the document No.3, the aforementioned problemconcerning the difficulty of the threshold value determination isresolved by calculating 3-D coordinates without using the disparities,in which parameters which represent relative positions of the imagepickup devices and angles between the optical axes of the image pickupdevices are calculated using images of a predetermined pattern picked upby the image pickup devices, instead of controlling or measuring therelative positions of the image pickup devices and angles between theoptical axes of the image pickup devices. Further, with regard to theaforementioned problem concerning the point correspondence search, errorrate in the point correspondence search is reduced in the method of thedocument No.3, by shooting the subject in the centers of the images bythe image pickup devices and executing the point correspondence searchstarting from the center of the images with high priority. Furthermore,even in the case where the false correspondence occurred in the pointcorrespondence search, parts which have dropped out from the extractionor parts which have been incorrectly extracted are corrected by theoutline restoration on the assumption of continuity. However, accordingto the method, all the image pickup devices have to be correctlycontrolled to shoot the subject in the centers of images. Further, inthe case where the background has a complex scene, error rate in thepoint correspondence search increases and thus the outline restorationis necessitated to be executed considerably oftener, causing difficultyin the subject image extraction especially when the subject has acomplex outline.

SUMMARY OF THE INVENTION

It is therefore the primary object of the present invention to provide asubject image extraction device by which an image of a desired subjectcan be extracted from an inputted image, without needing preparation ofthe information concerning the subject or the image pickup device.

Another object of the present invention is to provide a subject imageextraction device by which an image of a desired subject can beextracted from an inputted image, with easy setting of a threshold valuefor distinguishing between the background and the foreground.

Another object of the present invention is to provide a subject imageextraction device by which an image of a desired subject can beextracted from an inputted image, without needing precise control ormeasurement of the positions and attitudes of image pickup devices.

Another object of the present invention is to provide a subject imageextraction device by which 3-D coordinates corresponding to a point inan inputted image can be accurately obtained and thereby subject imageextraction can be executed with higher accuracy.

In accordance with the present invention, there is provided a subjectimage extraction device comprising an image storage means, a featurepoint extraction means, a parameter calculation means, a 3-D coordinatescalculation means, a 3-D coordinates determination means, a judgmentmeans, an outline point extraction means, and a subject extractionmeans. The image storage means includes main image storage section forstoring images which have been picked up by a main image pickup sectionfor picking up a main image from which an image of a subject isextracted, and two or more subsidiary image storage sections for storingimages which have been picked up by two or more subsidiary image pickupsections for picking up subsidiary images to be referred to in subjectimage extraction. The feature point extraction means includes three ormore feature point extraction sections for extracting feature pointcoordinates in the images which have been stored in each of the imagestorage sections. The parameter calculation means calculates parameterswhich represent relative positions and attitudes of the main imagepickup section and the subsidiary image pickup sections, using thefeature point coordinates in the images of a 3-D object on which apredetermined pattern is drawn which have been picked up by the mainimage pickup section and the subsidiary image pickup sections. The 3-Dcoordinates calculation means includes two or more 3-D coordinatescalculation sections for calculating 3-D candidate coordinates which areconsidered to correspond to the feature point coordinates of a featurepoint in the main image, using the feature point coordinates in the mainimage, the feature point coordinates in the subsidiary images, and theparameters obtained by the parameter calculation means. The 3-Dcoordinates determination means determines genuine 3-D coordinates whichcorrespond to the feature point in the main image, using the 3-Dcandidate coordinates calculated by the 3-D coordinates calculationsections in the 3-D coordinates calculation means. The judgment meansjudges whether or not the genuine 3-D coordinates corresponding to thefeature point in the main image is in a designated 3-D space domain, andaccepts the feature point if the genuine 3-D coordinates correspondingto the feature point is judged to be in the designated 3-D space domain.The outline point extraction means extracts subject outline points froma plurality of feature points which have been accepted by the judgmentmeans. And the subject extraction means extracts part of the main imagewhich is surrounded by the subject outline points as the subject image.

Preferably, the feature point extraction sections in the feature pointextraction means extract edge-like parts in the images where largechanges occur in color intensity or luminance as the feature pointcoordinates, in the extraction of the feature point coordinates from themain image or the subsidiary images.

Preferably, the parameter calculation means calculates eleven or moreparameters per one image pickup section.

Preferably, the 3-D coordinates determination means determines thegenuine 3-D coordinates by selecting a group of 3-D coordinates fromeach of the 3-D candidate coordinates calculated by each of the 3-Dcoordinates calculation sections so that the sum of distances betweenthe 3-D coordinates included in the group will become the smallest anddefining the genuine 3-D coordinates by the average of the 3-Dcoordinates included in the group.

Preferably, the subject image extraction device further comprises animage pickup means including the main image pickup section and the twoor more subsidiary image pickup sections.

Preferably, the subject image extraction device further comprises asubject display means for displaying the part of the main image whichhas been extracted by the subject extraction means.

Preferably, the number of the image pickup sections is three.

In accordance with another aspect of the present invention, there isprovided a subject image extraction device comprising an image storagemeans, a feature point neighborhood extraction means, a parametercalculation means, a corresponding point determination means, a 3-Dcoordinates calculation means, a 3-D coordinates determination means, ajudgment means, an outline point extraction means, and a subjectextraction means. The image storage means includes main image storagesection for storing images which have been picked up by a main imagepickup section for picking up a main image from which an image of asubject is extracted, and two or more subsidiary image storage sectionsfor storing images which have been picked up by two or more subsidiaryimage pickup sections for picking up subsidiary images to be referred toin subject image extraction. The feature point neighborhood extractionmeans includes three or more feature point neighborhood extractionsections for extracting feature point coordinates in the images whichhave been stored in each of the image storage sections and extractingcolor/luminance information of pixels in the neighborhood of the featurepoint coordinates. The parameter calculation means calculates parameterswhich represent relative positions and attitudes of the main imagepickup section and the subsidiary image pickup sections, using thefeature point coordinates in the images of a 3-D object on which apredetermined pattern is drawn which have been picked up by the mainimage pickup section and the subsidiary image pickup sections. Thecorresponding point determination means includes two or morecorresponding point determination sections for selecting a group ofcandidate corresponding points in the subsidiary image which areconsidered to correspond to the feature point in the main image, fromthe feature points in the subsidiary images, using the feature pointcoordinates in the main image and the subsidiary image and thecolor/luminance information of pixels in the neighborhood of the featurepoint coordinates which have been extracted by the feature pointneighborhood extraction means.

The 3-D coordinates calculation means includes two or more 3-Dcoordinates calculation sections for calculating 3-D candidatecoordinates which are considered to correspond to a feature point in themain image, using the feature point coordinates in the main image, thegroup of candidate corresponding points, and the parameters obtained bythe parameter calculation means. The 3-D coordinates determination meansdetermines genuine 3-D coordinates which correspond to the feature pointin the main image, using the 3-D candidate coordinates calculated by the3-D coordinates calculation sections in the 3-D coordinates calculationmeans.

The judgment means judges whether or not the genuine 3-D coordinatescorresponding to the feature point in the main image is in a designated3-D space domain, and accepts the feature point if the genuine 3-Dcoordinates corresponding to the feature point is judged to be in thedesignated 3-D space domain. The outline point extraction means extractssubject outline points from a plurality of feature points which havebeen accepted by the judgment means. And the subject extraction meansextracts part of the main image which is surrounded by the subjectoutline points as the subject image.

Preferably, the feature point neighborhood extraction sections in thefeature point extraction means extract edge-like parts in the imageswhere sudden changes occur in color intensity or luminance as thefeature point coordinates, in the extraction of the feature pointcoordinates from the main image or the subsidiary images.

Preferably, the parameter calculation means calculates eleven or moreparameters per one image pickup section.

Preferably, the 3-D coordinates determination means determines thegenuine 3-D coordinates by selecting a group of 3-D coordinates fromeach of the 3-D candidate coordinates calculated by each of the 3-Dcoordinates calculation sections so that the sum of distances betweenthe 3-D coordinates included in the group will become the smallest anddefining the genuine 3-D coordinates by the average of the 3-Dcoordinates included in the group.

Preferably, the subject image extraction device further comprises animage pickup means including the main image pickup section and the twoor more subsidiary image pickup sections.

Preferably, the subject image extraction device further comprises asubject display means for displaying the part of the main image whichhas been extracted by the subject extraction means.

Preferably, the number of the image pickup sections is three.

BRIEF DESCRIPTION OF THE DRAWINGS

The objects and features of the present invention will become moreapparent from the consideration of the following detailed descriptiontaken in conjunction with the accompanying drawings, in which:

FIG. 1 is a block diagram showing a subject image extraction deviceaccording to an embodiment of the present invention;

FIG. 2 is a schematic diagram showing preferred arrangement of imagepickup sections 1 a, 1 b and 1 c of the subject image extraction deviceof FIG. 1;

FIG. 3 is a perspective view showing an example of a 3-D object on whicha predetermined pattern is drawn which is used in the embodiment of FIG.1;

FIG. 4A is a schematic diagram showing a main image used in theembodiment of FIG. 1;

FIG. 4B is a schematic diagram showing a subsidiary image used in theembodiment of FIG. 1;

FIG. 5 is a schematic diagram showing the principle of triangularsurveying;

FIG. 6 is a schematic diagram showing a 3-D coordinates determinationprocess of the subject image extraction device of FIG. 1;

FIG. 7 is a schematic diagram explaining ‘block matching’ used in theembodiment of FIG. 8; and

FIG. 8 is a block diagram showing a subject image extraction deviceaccording to another embodiment of the present invention.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Referring now to the drawings, a description will be given in detail ofpreferred embodiments in accordance with the present invention.

FIG. 1 is a block diagram showing a subject image extraction deviceaccording to an embodiment of the present invention.

The subject image extraction device of FIG. 1 comprises an image pickupunit 1, an image storage unit 2, a feature point extraction unit 3, aparameter calculation unit 4, a 3-D coordinates calculation unit 5, a3-D coordinates determination unit 6, an outline point extraction unit7, and a subject display unit 8.

The image pickup unit 1 is composed of an image pickup section 1 a forpicking up a main image to be mainly used in a subject image extractionprocess which will be described below, and image pickup sections 1 b and1 c for picking up subsidiary images to be referred to in the subjectimage extraction process. The image pickup sections 1 a, 1 b and 1 c arepreliminarily used to shoot a 3-D object 21 on which a predeterminedpattern is drawn in order to obtain parameters 400 which will bedescribed below, and are used to shoot a subject 22 whose image will beextracted using the parameters 400 etc.

The image storage unit 2 is composed of image storage sections 2 a, 2 band 2 c. The image storage sections 2 a, 2 b and 2 c store images 10 a,10 b and 10 c of the 3-D object 21 and images 20 a, 20 b and 20 c of thesubject 22 which have been picked up by the image pickup sections 1 a, 1b and 1 c, respectively.

The feature point extraction unit 3 is composed of feature pointextraction sections 3 a, 3 b and 3 c. The feature point extractionsections 3 a, 3 b and 3 c extract coordinates 30 a, 30 b and 30 c of aplurality of feature points from the images 10 a, 10 b and 10 c whichhave been stored in the image storage sections 2 a, 2 b and 2 crespectively, and extract coordinates 50 a, 50 b and 50 c of a pluralityof feature points from the images 20 a, 20 b and 20 c which have beenstored in the image storage sections 2 a, 2 b and 2 c respectively.

The parameter calculation unit 4 calculates and stores the parameters400 which represent relative positions and attitudes of the main imagepickup section 1 a and the subsidiary image pickup sections 1 b and 1 c,using the feature point coordinates 30 a, 30 b and 30 c which have beenextracted by the feature point extraction sections 3 a, 3 b and 3 crespectively.

The 3-D coordinates calculation unit 5 is composed of 3-D coordinatescalculation sections 5 b and 5 c. The 3-D coordinates calculationsection 5 b calculates 3-D candidate coordinates 60 b using theparameters 400 and the feature point coordinates 50 a and 50 b whichhave been extracted by the feature point extraction sections 3 a and 3b. The 3-D coordinates calculation section 5 c calculates 3-D candidatecoordinates 60 c using the parameters 400 and the feature pointcoordinates 50 a and 50 c which have been extracted by the feature pointextraction sections 3 a and 3 c. Incidentally, the 3-D candidatecoordinates 60 b is a group of 3-D coordinates (i.e. a group of 3-Dvectors each of which is representing 3-D coordinates) which areconsidered to be candidates for genuine 3-D coordinates which correspondto the feature point (2-D) coordinates 50 a in the main image 20 a whichhave been extracted by the feature point extraction section 3 a.Similarly, the 3-D candidate coordinates 60 c is a group of 3-Dcoordinates (i.e. a group of 3-D vectors each of which is representing3-D coordinates) which are considered to be candidates for genuine 3-Dcoordinates which correspond to the feature point (2-D) coordinates 50 ain the main image 20 a which have been extracted by the feature pointextraction section 3 a.

The 3-D coordinates determination unit 6 determines genuine 3-Dcoordinates 700 which correspond to the feature point coordinates 50 ain the main image 20 a, using the 3-D candidate coordinates 60 b and the3-D candidate coordinates 60 c.

The outline point extraction unit 7 judges whether the feature point 50a is a point in the subject 22 or not, using the 3-D coordinates 700,and extracts subject outline point coordinates 800. Here, the subjectoutline point coordinates 800 is a group of 2-D coordinates (i.e. agroup of 2-D vectors each of which is representing 2-D coordinates)which correspond to a plurality of subject outline points (i.e. featurepoints 50 a which exist on the outline of the subject 22).

The subject display unit 8 displays part of the image 20 a which aresurrounded by the subject outline points, using the subject outlinepoint coordinates 800 extracted by the outline point extraction unit 7.

The feature point extraction unit 3, the parameter calculation unit 4,the 3-D coordinates calculation unit 5, the 3-D coordinatesdetermination unit 6, and the outline point extraction unit 7 arerealized, for example, by a microprocessor unit which is composed of aCPU, ROM (Read Only Memory), RAM (Random Access Memory), etc., andnecessary software. The image storage unit 2 is realized by one or morestorage devices such as an HDD (Hard Disk Drive), an MO (Magneto-Opticaldisk), etc. The subject display unit 8 is realized, for example, by adisplay unit such as an LCD (Liquid Crystal Display), etc. and amicroprocessor unit.

In the following, the operation of the subject image extraction deviceof FIG. 1 will be described, in which both a point itself andcoordinates of the point are expressed by the same word ‘point’ forbrevity.

In this embodiment, the parameters 400 which represent relativepositions and attitudes of the image pickup sections 1 a, 1 b and 1 care needed to be calculated by the parameter calculation unit 4 first ofall, using the coordinates 30 a, 30 b and 30 c of the feature points inthe images 10 a, 10 b and 10 c of the 3-D object 21.

For the calculation of the parameters 400, the image pickup sections 1a, 1 b and 1 c respectively pick up images 10 a, 10 b and 10 c of the3-D object 21 on which a predetermined pattern (which is shown in FIG.3, for example) is drawn.

The optical axes of the image pickup sections 1 a, 1 b and 1 c may beparallel, or may have convergence angles between them. Referring to FIG.2, the main image pickup section 1 a, for picking up the main image 10 ais placed in the middle of the three image pickup sections 1 a, 1 b and1 c. Such arrangement is advantageous since occlusion can be madesmaller.

The images 10 a, 10 b and 10 c of the 3-D object 21 on which thepredetermined pattern is drawn picked up by the image pickup sections 1a, 1 b and 1 c are stored in the image storage sections 2 a, 2 b and 2 crespectively, and then supplied to the feature point extraction sections3 a, 3 b and 3 c in the feature point extraction unit 3, respectively.

After the images 10 a, 10 b and 10 c are supplied to the feature pointextraction unit 3, six or more feature points P0, P1, P2, P3, P4, P5, .. . are chosen so that all the feature points will not be in one planeof the 3-D object 21, more concretely, so that a 3-D coordinate systemcan be at least generated using four points out of the six or morefeature points P0, P1, P2, P3, P4, P5, . . . Subsequently, coordinates30 a, 30 b and 30 c of the feature points P0, P1, P2, P3, P4, P5, . . .in the images 10 a, 10 b and 10 c are extracted by the feature pointextraction sections 3 a, 3 b and 3 c, respectively. Here, each of thecoordinates 30 a, 30 b and 30 c is a group of 2-D coordinates(corresponding to the feature points P0, P1, P2, P3, P4, P5, . . . ) inthe image 10 a, 10 b, 10 c, respectively. The choice of the featurepoints can be done by various kinds of methods, for example, byinstruction of the operator of the device who is watching the images 10a, 10 b and 10 c, or by automatic extraction by means of imageprocessing such as pattern matching etc.

Then, the parameter calculation unit 4 calculates the parameters 400using the coordinates 30 a, 30 b and 30 c of the feature points P0, P1,P2, P3, P4, P5, . . . , and the calculated parameters 400 are stored inthe parameter calculation unit 4.

The calculation of the parameters 400 can be executed based on a methodwhich is disclosed in Japanese Patent Application No.HEI7-206932. In thefollowing, the method for the calculation of the parameters 400 will bedescribed.

First, a 3-D coordinate system is defined so that coordinates of thefeature points P0, P1, P2 and P3 (shown in FIG. 3) in the 3-D coordinatesystem will be (0, 0, 0), (1, 0, 0), (0, 1, 0) and (0, 0, 1)respectively. Here, the 3-D coordinate system is not needed to be arectangular coordinate system with rectangular axes as is defined by theP0, P1, P2 and P3 which are shown in FIG. 3, and generally, the 3-Dcoordinate system may be an oblique coordinate system which is definedby feature points P0, P1, P2 and P3 which exist on oblique axes.Hereafter, coordinates of the feature points P4 and P5 in the 3-Dcoordinate system will be expressed as (X₄, Y₄, Z₄) and (X₅, Y₅, Z₅)respectively.

Incidentally, 2-D coordinates of the feature points P0, P1, P2, P3, P4and P5 in the image 10 a will be hereafter expressed as

(u _(i) ^(c) ,v _(i) ^(c))(i=0, . . . ,5),

and coordinates of the feature points P0, P1, P2, P3, P4 and P5 in theimage 10 b will be hereafter expressed as

(u _(i) ^(l) , v _(i) ^(l))(i=0, . . . ,5),

and coordinates of the feature points P0, P1, P2, P3 P4 and P5 in theimage 10 c will be hereafter expressed as

(u _(i) ^(r) , v _(i) ^(r))(i=0, . . . ,5),

Here, using the coordinates:

(u _(i) ^(c) , v _(i) ^(c)),

of the feature points in the image 10 a and the 3-D coordinates (X_(k),Y_(k), Z_(k)) (k=4, 5) of the feature points P4 and P5, and using thefollowing parameters:

c _(j) ^(c)(j=1, . . . ,3),

which represent the attitude of the main image pickup section 1 a, thefollowing equations (1), (2), (3) and (4) are derived.

 {X4(u ₄ ^(c) −u ₁ ^(c))}c ₁ ^(c) +{Y4(u ₄ ^(c) −u ₂ ^(c))}c ₂ ^(c)+{Z4(u ₄ ^(c) −u ₃ ^(c))}c ₃ ^(c)=(1−X4−Y4−Z4)(u ₀ ^(c) −u ₄ ^(c))  (1)

{X4(v ₄ ^(c) −v ₁ ^(c))}c ₁ ^(c) +{Y4(v ₄ ^(c) −v ₂ ^(c))}c ₂ ^(c)+{Z4(v ₄ ^(c) −v ₃ ^(c))}c ₃ ^(c)=(1−X4−Y4−Z4)(v ₀ ^(c) −v ₄ ^(c))  (2)

{X5(u ₅ ^(c) −u ₁ ^(c))}c ₁ ^(c) +{Y5(u ₅ ^(c) −u ₂ ^(c))}c ₂ ^(c)+{Z5(u ₅ ^(c) −u ₃ ^(c))}c ₃ ^(c)=(1−X5−Y5−Z5)(u ₀ ^(c) −u ₅ ^(c))  (3)

{X5(v ₅ ^(c) −v ₁ ^(c))}c ₁ ^(c) +{Y5(v ₅ ^(c) −v ₂ ^(c))}c ₂ ^(c)+{Z5(v ₅ ^(c) −v ₃ ^(c))}c ₃ ^(c)=(1−X5−Y5−Z5)(v ₀ ^(c) −v ₅ ^(c))  (4)

The above parameters:

c _(j) ^(c)(j=1, . . . ,3),

are elaborated on in A. Marugame et al., “Structure Recovery from ScaledOrthographic and Perspective Views,” Proceedings IEEE ICIP-96, vol.2,pp.851-854 (1996), and are representing the strain(extension/contraction) of three axes of the 3-D coordinate system dueto projection of the 3-D coordinate system onto 2-D plane by the imagepickup section 1 a. The parameters can be obtained from the equations(1), (2), (3) and (4), using, for example, method of least squares,which is described for example in a document: K. Kanatani, “Imageunderstanding”, Morikita Shuppan, Japan (1990).

Parameters:

c _(j) ^(l)(j=1, . . . ,3),

which represent the attitude of the subsidiary image pickup section 1 band parameters:

 c _(j) ^(r)(j=1, . . . ,3),

which represent the attitude of the subsidiary image pickup section 1 ccan be obtained by the same method as the case of the main image pickupsection 1 a.

The parameters:

c _(j) ^(c) , c _(j) ^(l) , c _(j) ^(r)(j=1, . . . ,3),

which have been obtained by the above calculation and the followingcoordinates:

(u _(i) ^(c) ,v _(i) ^(c)), (u _(i) ^(l) ,v _(i) ^(l)), (u _(i) ^(r) ,v_(i) ^(r))(i=0, . . . ,3),

are stored in the parameter calculation unit 4 as the parameters 400.Here, the number of parameters included in the parameters 400 per oneimage pickup section (1 a, 1 b or 1 c) is 11. It is generally known that11 parameters per one image pickup device are enough for expressingrelative positions and attitudes of a plurality of image pickup devices.

Now that the parameters 400 representing relative positions andattitudes of the image pickup sections 1 a, 1 b and 1 c have beenobtained and stored in the parameter calculation unit 4, extraction ofdesired subject in the main image 20 a can be executed. In thefollowing, the subject image extraction process will be described indetail.

The subject 22 is shot by the image pickup sections 1 a, 1 b and 1 c ofthe image pickup unit 1 and images 20 a, 20 b and 20 c are obtainedrespectively.

The images 20 a, 20 b and 20 c are stored in the image storage sections2 a, 2 b and 2 c of the image storage unit 2, respectively.

From the images 20 a, 20 b and 20 c which have been stored in the imagestorage sections 2 a, 2 b and 2 c respectively, feature pointcoordinates 50 a, 50 b and 50 c are respectively extracted by thefeature point extraction sections 3 a, 3 b and 3 c of the feature pointextraction unit 3.

The extraction of the feature points may be done by various kinds ofmethods, for example, by instruction of the operator of the device whois watching the images 20 a, 20 b and 20 c, or by automatic extractionusing differential filters such as Sobel filter, Laplacian filter, etc.The extraction of feature points 50 a are usually executed in units ofpixels in the main image 20 a. In the case where differential filtersare utilized for the aforementioned automatic extraction of the featurepoints 50 a, edge-like parts in the main image 20 a where large changesoccur in color intensity (in cases of color image) or luminance (incases of black and white image) are extracted as the feature points 50a, and thereby a few of all the pixels in the main image 20 a areextracted as the feature points 50 a by the feature point extractionsection 3 a, for example. The same goes for the extraction of featurepoints 50 b and 50 c from the images 20 b and 20 c.

A plurality of feature points 50 a are extracted one after another bythe feature point extraction section 3 a in the feature point extractionunit 3, and each feature point 50 a will be adopted as being a point onthe outline of the subject 22 (i.e. adopted as a ‘subject outlinepoint’) or rejected as not being a subject outline point, by theprocesses which will be described below. Finally, a group of subjectoutline points which are obtained as above will be used for extractionof the subject image.

The 3-D coordinates calculation section 5 b in the 3-D coordinatescalculation unit 5 calculates the 3-D candidate coordinates 60 b usingthe parameters 400 and the feature points 50 a and 50 b, and the 3-Dcoordinates calculation section 5 c calculates the 3-D candidatecoordinates 60 c using the parameters 400 and the feature points 50 aand 50 c. As mentioned above, the 3-D candidate coordinates 60 b are agroup of 3-D coordinates (i.e. a group of 3-D vectors each of which isrepresenting 3-D coordinates) which are considered to be candidates forgenuine 3-D coordinates which correspond to the feature point (2-D)coordinates 50 a in the main image 20 a which have been extracted by thefeature point extraction section 3 a, and the 3-D candidate coordinates60 c is a group of 3-D coordinates (i.e. a group of 3-D vectors each ofwhich is representing 3-D coordinates) which are considered to becandidates for genuine 3-D coordinates which correspond to the featurepoint (2-D) coordinates 50 a in the main image 20 a which have beenextracted by the feature point extraction section 3 a.

In the following, the operation of the 3-D coordinates calculationsection 5 b will be described in detail.

The 3-D coordinates calculation section 5 b chooses one of the featurepoints 50 a one after another, and the following processes are executedto each of the feature points 50 a. When 2-D coordinates of the featurepoint 50 a in the image 20 a is expressed as (u^(c), v^(c)) as shown inFIG. 4A, the position of a feature point 501 in the image 20 b whichcorresponds to the feature point 50 a in the image 20 a is restricted ona line 502 shown in FIG. 4B (which is called an epipolar line). Theepipolar line 502 is determined according to the relation between thepositions of the image pickup sections 1 a and 1 b. Therefore, the 3-Dcoordinates calculation section 5 b calculates the epipolar line 502using the following parameters which are included in the parameters 400.

i c_(j) ^(c) ,c _(j) ^(l)(j=1, . . . ,3), (u _(i) ^(c) ,v _(i) ^(c)), (u_(i) ^(l) ,v _(i) ^(l))(i=0, . . . ,3),

The calculation of the epipolar line 502 can be executed according tothe aforementioned document No.3 or a document: The proceedings of the1st Image Media Processing Symposium (1996), pages 15-16, as follows.

First, using the parameters:

(u _(i) ^(c) ,v _(i) ^(c))(i=0, . . . ,3), c _(j) ^(c)(j=1, . . . ,3),(u ^(c) ,v ^(c)),

the following primary intermediate parameters t₁₁, t₁₂, t₁₃, t₂₁, t₂₂,t₂₃, d₁ and d₂ are obtained.

t11={c ₁ ^(c)(u ^(c) −u ₁ ^(c))−(u ^(c) −u ₀ ^(c))}  (5)

t12={c ₂ ^(c)(u ^(c) −u ₂ ^(c))−(u ^(c) −u ₀ ^(c))}  (6)

t13={c ₃ ^(c)(u ^(c) −u ₃ ^(c))−(u ^(c) −u ₀ ^(c))}  (7)

t21={c ₁ ^(c)(v ^(c) −v ₁ ^(c))−(v ^(c) −v ₀ ^(c))}  (8)

t22={c ₂ ^(c)(v ^(c) −v ₂ ^(c))−(v ^(c) −v ₀ ^(c))}  (9)

t23={c ₃ ^(c)(v ^(c) −v ₃ ^(c))−(v ^(c) −v ₀ ^(c))}  (10)

d1=u ₀ ^(c) −u ^(c)  (11)

d2=v ₀ ^(c) −v ^(c)  (12)

Subsequently, using the primary intermediate parameters t₁₁, t₁₂, t₁₃,t₂₁, t₂₂, t₂₃, d₁ and d₂ and the following parameters:

(U _(i) ^(l) , V _(i) ^(l)) (i=0, . . . ,3), C _(j) ^(l)(j=1, . . . ,3),

the following secondary intermediate parameters U₀, U₁, V₀, V₁, S₀ andS₁ are obtained.

U ₀ =u ₀ ^(l)(t ₁₁ t ₂₂ −t ₁₂ t ₂₁)+(c ₁ ^(l) u ₁ ^(l) −u ₀ ^(l)) (t ₂₂d ₁ −t ₁₂ d ₂)+(c ₂ ^(l) u ₂ ^(l) −u ₀ ^(l))(t ₁₁ d ₂ −t ₂₁ d ₁)  (13)

U ₁=(c ₁ ^(l) u ₁ ^(l) −u ₀ ^(l))(t ₁₂ t ₂₃ −t ₁₃ t ₂₂)+(c ₂ ^(l) u ₂^(l) −u ₀ ^(l))(t ₁₃ t ₂₁ −t ₁₁ t ₂₃)+(c ₃ ^(l) u ₃ ^(l) −u ₀ ^(l))(t ₁₁t ₂₂ −t ₁₂ t ₂₁)  (14)

V ₀ =U ₀ ^(l)(t ₁₁ t ₂₂ −t ₁₂ t ₂₁)+(c ₁ ^(l) V ₁ ^(l) −V ₀ ^(l)) (t ₂₂d ₁ −t ₁₂ d ₂)+(c ₂ ^(l) v ₂ ^(l) −v ₀ ^(l))(t ₁₁ d ₂ −t ₂₁ d ₁)  (15)

V ₁=(c ₁ ^(l) v ₁ ^(l) −v ₀ ^(l))(t ₁₂ t ₂₃ −t ₁₃ t ₂₂)+(c ₂ ^(l) v ₂^(l) −v ₀ ^(l))(t ₁₃ t ₂₁ −t ₁₁ t ₂₃)+(c ₃ ^(l) v ₃ ^(l) −v ₀ ^(l))(t ₁₁t ₂₂ −t ₁₂ t ₂₁)  (16)

S ₀=(t ₁₁ t ₂₂ −t ₁₂ t ₂₁)+(c ₁ ^(l)−1)(t ₂₂ d ₁ −t ₁₂ d ₂)+(c ₂^(l)−1)(t ₁₁ d ₂ −t ₂₁ d ₁)  (17)

S ₁=(c ₁ ^(l)−1)(t ₁₂ t ₂₃ −t ₁₃ t ₂₂)+(c ₂ ^(l)−1)(t ₁₃ t ₂₁ −t ₁₁ t₂₃)+(c ₃ ^(l)−1)(t ₁₁ t ₂₂ −t ₁₂ t ₂₁)  (18)

Subsequently, using the secondary intermediate parameters U₀, U₁, V₀,V₁, S₀ and S₁, the parameters A, B and C of the epipolar line 502:Au+Bv+C=0 (Here, the (u, v) are 2-D coordinates in the image 20 b.) areobtained by the following equations (19)-(21). As mentioned above, theepipolar line 502 is a line in the image 20 b which restricts theposition of the feature point 501 in the image 20 b which corresponds tothe feature point 50 a in the image 20 a.

A=S ₁ V ₀ −S ₀ V ₁  (19)

B=−(S ₁ U ₀ −S ₀ U ₁)  (20)

C=U ₀ V ₁ −U ₁ V ₀  (21)

Now that the parameters A, B and C of the epipolar line 502 have beenobtained, feature points Q_(m): (u_(m), v_(m)) (m=1, . . . ) which existin the ε₁-neighborhood of the epipolar line 502 in the image 20 b can beexpressed by the following inequality. $\begin{matrix}{\frac{{{A\quad u_{m}} + {B\quad v_{m}} + C}}{\sqrt{A^{2} + B^{2}}} < ɛ_{1}} & (22)\end{matrix}$

Incidentally, the number of the feature points Q_(m) in theε₁-neighborhood of the epipolar line 502 is finite (for example, of theorder of 10), since feature points Q_(m) are usually defined in units ofpixels in the image 20 b. The above feature points Q_(m): (u_(m), v_(m))(m=1, . . . ) are candidate corresponding points in the image 20 b whichare considered to correspond to the feature point 50 a: (u_(c), v_(c))in the image 20 a.

After the feature points Q_(m): (u_(m), v_(m)) (m=1, . . . ) areobtained, the 3-D candidate coordinates 60 b:

(X _(m) ^(l) ,Y _(m) ^(l) ,Z _(m) ^(l)) (m=1, . . . ),

are calculated for each pair of the feature point 50 a and the featurepoint Q_(m) (m=1, . . . ).

The calculation of the 3-D candidate coordinates 60 b:

(X _(m) ^(l) ,Y _(m) ^(l) ,Z _(m) ^(l)) (m=1, . . . ),

can be executed based on the aforementioned Japanese Patent ApplicationNo.HEI7-206932 as follows, using the feature point 50 a: (u^(C), v^(c)),the feature points Q_(m): (u_(m), v_(m)) (m=1, . . . ), and thefollowing parameters.

(u _(i) ^(c) ,v _(i) ^(c)), (u _(i) ^(l) ,v _(i) ^(l))(i=0, . . . ,3), c_(j) ^(c) ,c _(j) ^(l)(j=1, . . . ,3),

First, referring to FIG. 5, a set of simultaneous equations whichrepresent a back projection line 503 of the feature point 50 a (u^(c),v^(c)):

{c ₁ ^(c)(u ^(c) −u ₁ ^(c))−(u ^(c) −u ₀ ^(c))}X _(m) ^(l) +{c ₂ ^(c)(u^(c) −u ₂ ^(c))−(u ^(c) −u ₀ ^(c))}Y _(m) ^(l) +{c ₃ ^(c)(u ^(c) −u ₃^(c))−(u ^(c) −u ₀ ^(c))}Z _(m) ^(l) =u ₀ ^(c) −u ^(c)  (23)

{c ₁ ^(c)(v ^(c) −v ₁ ^(c))−(v ^(c) −v ₀ ^(c))}X _(m) ^(l) +{c ₂ ^(c)(v^(c) −v ₂ ^(c))−(v ^(c) −v ₀ ^(c))}Y _(m) ^(l) +{c ₃ ^(c)(v ^(c) −v ₃^(c))−(v ^(c) −v ₀ ^(c))}Z _(m) ^(l) =v ₀ ^(c) −v ^(c)  (24)

and a set of simultaneous equations which represent a back projectionline 504 of the candidate corresponding point 51 b (Q_(m): (u_(m),v_(m))):

{c ₁ ^(l)(u _(m) −u ₁ ^(l))−(u _(m) −u ₀ ^(l))}X _(m) ^(l) +{c ₂ ^(l)(u_(m) −u ₂ ^(l))−(u _(m) −u ₀ ^(l))}Y _(m) ^(l) +{c ₃ ^(l))(u _(m) −u ₃^(l))−(u _(m) −u ₀ ^(l))}Z _(m) ^(l) =u ₀ ^(l) −u ^(l)  (25)

{c ₁ ^(l)(v _(m) −v ₁ ^(l))−(v _(m) −v ₀ ^(l))}X _(m) ^(l) +{c ₂ ^(l)(v_(m) −v ₂ ^(l))−(v _(m) −v ₀ ^(l))}Y _(m) ^(l) +{c ₃ ^(l)(v _(m) −v ₃^(l))−(v _(m) −v ₀ ^(l))}Z _(m) ^(l) =v ₀ ^(l) −v ^(l)  (26)

are derived.

According to the principle of triangular surveying, the 3-D candidatecoordinates 60 b:

(X _(m) ^(l) ,Y _(m) ^(l) ,Z _(m) ^(l))(m=1, . . . ),

can be obtained by solving the simultaneous equations (23). (24), (25)and (26). The solution:

(X _(m) ^(l) ,Y _(m) ^(l) ,Z _(m) ^(l)) (m=1, . . . ),

of the simultaneous equations (23), (24), (25) and (26) can be obtainedby means of method of least squares etc.

The 3-D candidate coordinates 60 b:

(X _(m) ^(l) ,Y _(m) ^(l) ,Z _(m) ^(l)) (m=1, . . . ),

obtained as above are sent to the 3-D coordinates determination unit 6,with the feature point coordinates 50 a (U^(c), V^(c)) attached theretoas a label.

Meanwhile, the 3-D coordinates calculation section 5 c obtains the 3-Dcandidate coordinates 60 c by the same method as the 3-D coordinatescalculation section 5 b, using the following parameters which areincluded in the parameters 400.

(u _(i) ^(c) ,v _(i) ^(c)), (u _(i) ^(r) ,v _(i) _(r))(i=0, . . . ,3),c_(j) ^(c) ,C _(j) ^(r)(j=1, . . . ,3),

The obtained 3-D candidate coordinates 60 c:

(X _(n) ^(r) ,Y _(n) ^(r) ,Z _(n) ^(r)) (n =1, . . . . ),

are sent to the 3-D coordinates determination unit 6, with the featurepoint coordinates 50 a (u^(c), v^(c)) attached thereto as a label.

The 3-D coordinates determination unit 6, which has been supplied withthe 3-D candidate coordinates 60 b and 60 c from the 3-D coordinatescalculation sections 5 b and 5 c, determines the genuine 3-D coordinates700 using the 3-D candidate coordinates 60 b and 60 c which correspondto the same feature point 50 a: (U^(c), V^(c)).

In the following, the operation of the 3-D coordinates determinationunit 6 will be described in detail. Referring to FIG. 6, the featurepoint 50 a in the image 20 a is in a one-to-one correspondence with acorresponding point 61 on the subject 22 in the 3-D coordinate system.Thus, a 3-D candidate point 60 b which corresponds to a (2-D) candidatecorresponding point 51 b (Q_(m): (u_(m), v_(m))) in the image 20 b(Here, the candidate corresponding point 51 b in the image 20 b is apoint which is considered to correspond to the feature point 50 a in theimage 20 a.) and a 3-D candidate point 60 c which corresponds to a (2-D)candidate corresponding point 51 c in the image 20 c (Here, thecandidate corresponding point 51 c in the image 20 c is a point which isconsidered to correspond to the feature point 50 a in the image 20 a.)have to be the same point (i.e. the aforementioned corresponding point61), and thus have to have almost the same 3-D coordinates. Therefore,for all the combinations of the 3-D candidate points 60 b:

(X _(m) ^(l) ,Y _(m) ^(l) ,Z _(m) ^(l)) (m=1, . . . ),

and the 3-D candidate points 60 c:

(X _(n) ^(r) ,Y _(n) ^(r) ,Z _(n) ^(r))(n=1, . . . ),

, the following absolute norm d_(mn):

d _(mn)=(X _(m) ^(l) −X _(n) ^(r))²+(Y _(m) ^(l) −Y _(n) ^(r))₂+(Z _(m)^(l) −Z _(n) ^(r))²  (27)

or the following L² norm d_(mn):

d _(mn) =|X _(m) ^(l) −X _(n) ^(r) |+|Y _(m) ^(l) −Y _(n) ^(r) |+|Z _(m)^(l) −Z _(n) ^(r|)  (28 )

is calculated. Here, the norm d_(mn) represents the distance between the3-D candidate point 60 b and the 3-D candidate point 60 c. Then, if theminimun value d=d_(ij) (where d_(ij)=min(d_(mn))) is not larger than athreshold value ε₂, the feature point 50 a: (u^(c), v^(c)) is acceptedand the 3-D coordinates (X, Y, Z) of the point 61 are determined asfollows.

X=(X ₁ ^(l) +X _(j) ^(r))/2  (29)

Y=(Y _(i) ^(l) +Y _(j) ^(r))/2  (30)

Z=(Z _(i) ^(l) +Z _(j) ^(r))/2  (31)

And if the minimum value d=d_(ij) is larger than the threshold value ε₂,the feature point 50 a: (u^(c), v^(c)) is rejected or discarded as apoint out of the subject. The above (preliminary) judgment using thethreshold value ε₂ is executed in order to reject feature points 50 awhich have not been picked up by all of the image pickup sections 1 a, 1b and 1 c.

After the determination by the 3-D coordinates determination unit 6, thegenuine 3-D coordinates 700: (X, Y, Z) of the point 61 is outputted tothe outline point extraction unit 7, with the feature point coordinates50 a (u^(c), v^(c)) in the image 20 a attached thereto as a label.

The outline point extraction unit 7 first executes judgment whether thefeature point 50 a is a point in the subject 22 or not, in which theoutline point extraction unit 7 accepts the feature point 50 a: (u^(c),v^(c)) (which has been attached to the 3-D coordinates 700: (X, Y, Z) ofthe point 61) if the 3-D coordinates 700: (X, Y, Z) is within theconfines of a designated 3-D space domain which has been preliminarilydetermined or which is designated on the spot by the operator. Aplurality of feature points 50 a: (u^(c), v^(c)) and their 3-Dcoordinates 700: (X, Y, Z) are supplied to the outline point extractionunit 7 successively, and the outline point extraction unit 7 executesthe above judgment (whether the feature point 50 a: (u^(c), v^(c)) canbe accepted or not) successively. Then, from a plurality of featurepoints 50 a which have been accepted by the outline point extractionunit 7 and which exist on a horizontal line (i.e. a plurality ofaccepted feature points 50 a: (u^(c), v^(c)) which have the same Ycoordinate v^(c)), the outline point extraction unit 7 selects aleft-hand feature point (a feature point 50 a on the horizontal linewhose X coordinate u^(c) is the smallest) and a right-hand feature point(a feature point 50 a on the horizontal line whose X coordinate u^(c) isthe largest), and sends the left-hand feature points and the right-handfeature points corresponding to each Y coordinate v^(c) into the subjectdisplay unit 8 as the subject outline point coordinates 800.

Then, the subject display unit 8 displays part of the main image 20 awhich is surrounded by the subject outline point coordinates 800 (i.e.part of the image 20 a between the left-hand feature points and theright-hand feature points corresponding to each Y coordinate V^(c)),thereby the image of the subject 22 is extracted from the main image 20a and displayed on the subject display unit 8.

As described above, in the first embodiment, the main image pickingsection 1 a and the subsidiary image picking sections 1 b and 1 c areused to shoot the subject, and images 10 a, 10 b and 10 c of the 3-Dobject 21 on which a predetermined pattern is drawn are also picked upby the image picking sections 1 a, 1 b and 1 c in order to obtainparameters 400 which represent relative positions and attitudes of theimage pickup sections 1 a, 1 b and 1 c. In the embodiment, 11 parametersper one image pickup section (generally known to be enough forexpressing relative positions and attitudes of a plurality of imagepickup sections) are obtained by the parameter calculation unit 4. Fromthe images 20 a, 20 b and 20 c in which the subject 22 has been pickedup, the feature points 50 a, 50 b and 50 c are extracted by the featurepoint extraction unit 3, and the genuine 3-D coordinates 700corresponding to each feature point 50 a in the main image 20 a areobtained by the 3-D coordinates calculation unit 5 and the 3-Dcoordinates determination unit 6 using the parameters 400 and thefeature points 50 b and 50 c. In the outline point extraction unit 7,judgment whether each feature point 50 a is a point in the subject 22 ornot is executed using the 3-D coordinates 700 corresponding to thefeature point 50 a, in which a feature point 50 a is accepted if the 3-Dcoordinates 700 corresponding to the feature point 50 a is within theconfines of a designated 3-D space domain which has been preliminarilydetermined or which is designated on the spot by the operator, andotherwise, the feature point 50 a is rejected as not a point in thesubject 22, and the subject outline point coordinates 800 are extractedfrom a plurality of feature points 50 a which have been accepted. Then,part of the main image 20 a which is surrounded by the subject outlinepoint coordinates 800 is extracted and displayed by the subject displayunit 8, thereby the subject image extraction from the main image 20 a iscompleted.

Therefore, according to the first embodiment, an image of a desiredsubject can be extracted from the main image 20 a, using the 3-Dcoordinates calculated for each extracted feature point 50 a andexecuting the judgment with regard to the designated 3-D space domainwhich has been preliminarily determined or which is designated on thespot by the operator, without needing preparation of the informationconcerning the subject or the image pickup devices.

Further, according to the first embodiment, subject image extraction canbe executed with easy setting of a threshold value for distinguishingbetween the background and the foreground, since 3-D coordinatescorresponding to a feature point 50 a in the image 20 a areautomatically calculated using the parameters 400 etc. and the operatorof the device can set the threshold value by measurable value such asdistance or by designating the 3-D space domain.

Further, according to the first embodiment, subject image extraction canbe executed without needing precise control or measurement of thepositions and attitudes of the image pickup devices, since theparameters 400 representing relative positions and attitudes of theimage pickup sections 1 a, 1 b and 1 c can be automatically obtainedusing the images 10 a, 10 b and 10 c of the 3-D object 21 on which apredetermined pattern is drawn.

Further, according to the first embodiment, accuracy of subject imageextraction can be remarkably improved, since the 3-D coordinates 700which correspond to each extracted feature point 50 a in the main image20 a can be accurately obtained using the parameters 400 calculated fromthe images 10 a, 10 b, 10 c picked up by the image pickup sections 1 a.1 b and 1 c, and the feature points 50 b and 50 c extracted from thesubsidiary images 20 b and 20 c. The accuracy of subject imageextraction is increased with a synergistic effect by the extraction ofthe feature points 50 a, 50 b and 50 c from the aforementioned edge-likeparts of the images 20 a, 20 b and 20 c and by the accurate calculationof the genuine 3-D coordinates 700 and by the designation of thedesignated 3-D space domain.

Incidentally, although three image pickup sections (i.e. one main imagepickup section 1 a and two subsidiary image pickup sections 1 b and 1 c)were used in the image pickup unit 1 in the above embodiment, it is alsopossible to use four or more image pickup sections (i.e. three or moresubsidiary image pickup sections). In the case where N pieces ofsubsidiary image pickup sections are used, N+1 pieces of image storagesections, N+1 pieces of feature point extraction sections, and N piecesof 3-D coordinates calculation sections are used. In this case, (N+1)×11parameters (i.e. 11 parameters per image pickup sections) are calculatedby the parameter calculation unit 4 and N groups of 3-D candidatecoordinates (60 b, 60 c, . . . ) are obtained by the 3-D coordinatescalculation unit 5, and in the 3-D coordinates determination unit 6, thesum of norms d_(mn) (with regard to every pair of subsidiary images) areused instead of the norm d_(mn) of the equation (27) or (28), and theaverages in the equations (29) through (31) are replaced by averagesdivided by N (not by 2).

FIG. 8 is a block diagram showing a subject image extraction deviceaccording to the second embodiment of the present invention.

The subject image extraction device of FIG. 8 comprises an image pickupunit 1, an image storage unit 2, a feature point neighborhood extractionunit 33, a parameter calculation unit 4, a corresponding pointdetermination unit 9, a 3-D coordinates calculation unit 55, a 3-Dcoordinates determination unit 6, an outline point extraction unit 7,and a subject display unit 8.

The image pickup unit 1 is composed of an image pickup section 1 a forpicking up the main image to be mainly used in the subject imageextraction process, and image pickup sections 1 b and 1 c for picking upsubsidiary images to be referred to in the subject image extractionprocess. The image pickup sections 1 a, 1 b and 1 c are preliminarilyused to shoot a 3-D object 21 on which a predetermined pattern is drawnin order to obtain the parameters 400, and are used to shoot a subject22 whose image will be extracted using the parameters 400 etc.

The image storage unit 2 is composed of image storage sections 2 a, 2 band 2 c. The image storage sections 2 a, 2 b and 2 c store images 10 a,10 b and 10 c of the 3-D object 21 and images 20 a, 20 b and 20 c of thesubject 22 which have been picked up by the image pickup sections 1 a, 1b and 1 c, respectively.

The feature point neighborhood extraction unit 33 is composed of featurepoint neighborhood extraction sections 33 a, 33 b and 33 c. The featurepoint neighborhood extraction sections 33 a, 33 b and 33 c extractcoordinates 30 a, 30 b and 30 c of a plurality of feature points fromthe images 10 a, 10 b and 10 c which have been stored in the imagestorage sections 2 a, 2 b and 2 c respectively, and extract coordinates50 a, 50 b and 50 c of a plurality of feature points from the images 20a, 20 b and 20 c which have been stored in the image storage sections 2a, 2 b and 2 c respectively, and further extract color/luminanceinformation 52 a, 52 b and 52 c of pixels in the neighborhood of thefeature points 50 a, 50 b and 50 c respectively.

The parameter calculation unit 4 calculates and stores the parameters400 which represent relative positions and attitudes of the main imagepickup section 1 a and the subsidiary image pickup sections 1 b and 1 c,using the feature point coordinates 30 a, 30 b and 30 c which have beenextracted by the feature point extraction sections 3 a, 3 b and 3 crespectively.

The corresponding point determination unit 9 is composed ofcorresponding point determination sections 9 b and 9 c. Thecorresponding point determination section 9 b selects a group ofcandidate corresponding points 90 b in the image 20 b (points in theimage 20 b which are considered to correspond to the feature point 50 ain the image 20 a) from the feature points 50 b, using the featurepoints 50 a and 50 b and the color/luminance information 52 a and 52 bof pixels in the neighborhood of the feature points 50 a and 50 b whichhave been extracted by the feature point neighborhood extractionsections 33 a and 33 b and using the parameters 400. The correspondingpoint determination section 9 c selects a group of candidatecorresponding points 90 c in the image 20 c (points in the image 20 cwhich are considered to correspond to the feature point 50 a in theimage 20 a) from the feature points 50 c, using the feature points 50 aand 50 c and the color/luminance information 52 a and 52 c of pixels inthe neighborhood of the feature points 50 a and 50 c which have beenextracted by the feature point neighborhood extraction sections 33 a and33 c and using the parameters 400.

The 3-D coordinates calculation unit 55 is composed of 3-D coordinatescalculation sections 55 b and 55 c. The 3-D coordinates calculationsection 55 b calculates 3-D candidate coordinates 60 b using the featurepoint 50 a which has been extracted by the feature point neighborhoodextraction section 33 a, the candidate corresponding points 90 b whichhave been selected by the corresponding point determination section 9b,and the parameters 400. The 3-D coordinates calculation section 55 ccalculates 3-D candidate coordinates 60 c using the feature point 50 awhich has been extracted by the feature point neighborhood extractionsection 33 a, the candidate corresponding points 90 c which have beenselected by the corresponding point determination section 9 c, and theparameters 400.

The 3-D coordinates determination unit 6 determines genuine 3-Dcoordinates 700 which correspond to the feature point coordinates 50 ain the main image 20 a, using the 3-D candidate coordinates 60 b and the3-D candidate coordinates 60 c.

The outline point extraction unit 7 judges whether the feature point 50a is a point in the subject 22 or not, using the 3-D coordinates 700,and extracts subject outline point coordinates 800.

The subject display unit 8 displays part of the image 20 a which aresurrounded by the subject outline points, using the subject outlinepoint coordinates 800 extracted by the outline point extraction unit 7.

In the following, the operations of the corresponding pointdetermination unit 9 (which has been newly introduced into the secondembodiment) and the feature point neighborhood extraction unit 33 andthe 3-D coordinates calculation unit 55 (whose operations have beenmodified in the second embodiment) will be described in detail. Theoperations of the other blocks in FIG. 8 is the same as those of thefirst embodiment.

First, the operation of the feature point neighborhood extraction unit33 will be described. When the images 10 a, 10 b and 10 c of the 3-Dobject 21 on which the predetermined pattern is drawn are supplied tothe feature point neighborhood extraction unit 33, the feature pointneighborhood extraction sections 33 a, 33 b and 33 c operate in the sameway as the feature point extraction sections 3 a, 3 b and 3 c of thefirst embodiment and extract the coordinates 30 a, 30 b and 30 c of aplurality of feature points from the images 10 a, 10 b and 10 c andoutput the feature point coordinates 30 a, 30 b and 30 c to theparameter calculation unit 4, respectively. When the images 20 a, 20 band 20 c of the subject 22 are supplied to the feature pointneighborhood extraction unit 33, the feature point neighborhoodextraction sections 33 a, 33 b and 33 c extract the feature points 50 a,50 b and 50 c respectively in the same way as the feature pointextraction sections 3 a, 3 b and 3 c of the first embodiment, andfurther extract color/luminance information (color intensity informationin the case of color images or luminance information in the case ofblack and white images) 52 a, 52 b and 52 c of pixels in theneighborhood of the feature points 50 a, 50 b and 50 c respectively. Theneighborhood can be defined in various ways, however, a rectangle whosecenter is on the feature point is generally used for the neighborhood.

Next, the operation of the corresponding point determination unit 9 willbe described. The corresponding point determination section 9 b in thecorresponding point determination unit 9 extracts candidatecorresponding points 5 b (i.e. the Q_(m): (u_(m,) v_(m)) (m=1, . . . )which are obtained by the aforementioned inequality (22)) in the image20 b which corresponds to the feature point 50 a by calculating theepipolar line in the same way as the 3-D coordinates calculation section5 b of the first embodiment. Subsequently, the corresponding pointdetermination section 9 b executes block matching, which is known as animage processing technique, between the color/luminance information 52 aof the pixels in the neighborhood of the feature point 50 a and thecolor/luminance information 52 b of the pixels in the neighborhood of acandidate corresponding point 51 b. FIG. 7 is a schematic diagramexplaining the block matching. In the block matching between thecolor/luminance information 52 a and 52 b, two blocks of the same sizerespectively including the feature point 50 a and the candidatecorresponding point 51 b are formed first, and comparison of thecolor/luminance between two corresponding points in the two blocks isexecuted. Then, the difference of the color/luminance is added up withrespect to each point in the blocks (The sum is referred to as amatching error.). If the matching error is larger than a predeterminedthreshold value, the candidate corresponding point 51 b is rejected asan inadequate point (a point not corresponding to the feature point 50 a). After the rejection by the block matching, a group of candidatecorresponding points 90 b from which the inadequate points have beenremoved is sent to the 3-D coordinates calculation section 55 b in the3-D coordinates calculation unit 55. At this point, the candidatecorresponding points 90 b have been screened by the block matching, andthe number of the candidate corresponding points 90 b have beenconsiderably reduced due to the block matching. Meanwhile, thecorresponding point determination section 9 c determines a group ofcandidate corresponding points 90 c in the same way as the correspondingpoint determination section 9 b, and sends the group of candidatecorresponding points 90 c to the 3-D coordinates calculation section 55c in the 3-D coordinates calculation unit 55. Incidentally, although theabove corresponding point determination unit 9 used the parameters 400for calculating the epipolar line and obtaining the candidatecorresponding points, it is also possible to obtain candidatecorresponding points without using the parameters 400.

Lastly, the operation of the 3-D coordinates calculation unit 55 will bedescribed. The 3-D coordinates calculation section 55 b in the 3-Dcoordinates calculation unit 55 obtains almost all 3-D candidatecoordinates 60 b by the same operation as the 3-D coordinatescalculation section 5 b of the first embodiment. However, in the secondembodiment, the candidate corresponding points 90 b have been alreadyobtained by the corresponding point determination section 9 b, thereforethe 3-D coordinates calculation section 55 b does not calculate thecandidate corresponding points Q_(m): (u_(m), v_(m)) (m=1, . . . ) anduses the candidate corresponding points 90 b supplied by thecorresponding point determination section 9 b.

As described above, according to the second embodiment, as well as thesame effects as those of the first embodiment, screening ofcorresponding points is preliminarily executed by the correspondingpoint determination unit 9 by means of the block matching, and thusaccuracy of calculation and determination of the 3-D coordinates whichcorrespond to the feature point 50 a can be further improved, andcalculation time for the 3-D coordinates (which was the major portion ofcalculation time of the subject image extraction device) can beconsiderably shortened.

As set forth hereinbefore by the subject image extraction deviceaccording to the present invention, an image of a desired subject can beextracted from an inputted image with high accuracy, without needingpreparation of the information concerning the subject or the imagepickup devices, without needing precise control or measurement of thepositions and attitudes of the image pickup devices, and with easysetting of a threshold value for distinguishing between the backgroundand the foreground.

The subject image extraction devices which have been described referringto FIG. 1 and FIG. 8 can be applied to reduction of the amount of datawhich is transmitted between video phone etc. In the case of videophone, a speaker talking near the video phone (i.e. in the designated3-D space domain) can be extracted as the subject from an image, and theimage of the speaker only can be encoded and transmitted, thereby theamount of data can be considerably reduced, performance of video phonecan be improved, and privacy of speakers can be protected.

Incidentally, although the subject image extraction devices of FIG. 1and FIG. 8 included the image pickup unit 1, subject image extractioncan of course be executed by a subject image extraction devices which isnot provided with the image pickup unit 1 as long as images of a 3-Dobject and images including a subject which have been picked up by threeor more image pickup sections (devices) are supplied. Therefore, subjectimage extraction devices according to the present invention withoutimage pickup units are also possible. Similarly, subject imageextraction devices according to the present invention for subject imageextraction only (i.e. not for displaying the extracted subject image)are also possible. Description of such subject image extraction devicesis omitted for brevity.

While the present invention has been described with reference to theparticular illustrative embodiments, it is not to be restricted by thoseembodiments but only by the appended claims. It is to be appreciatedthat those skilled in the art can change or modify the embodimentswithout departing from the scope and spirit of the present invention.

What is claimed is:
 1. A subject image extraction device comprising: animage storage means including main image storage section for storingimages which have been picked up by a main image pickup section forpicking up a main image from which an image of a subject is extracted,and two or more subsidiary image storage sections for storing imageswhich have been picked up by two or more subsidiary image pickupsections for picking up subsidiary images to be referred to in subjectimage extraction; a feature point extraction means including three ormore feature point extraction sections for extracting feature pointcoordinates in the images which have been stored in each of the imagestorage sections; a parameter calculation means for calculatingparameters which represent relative positions and attitudes of the mainimage pickup section and the subsidiary image pickup sections, using thefeature point coordinates in the images of a 3-D object on which apredetermined pattern is drawn which have been picked up by the mainimage pickup section and the subsidiary image pickup sections; a 3-Dcoordinates calculation means including two or more 3-D coordinatescalculation sections for calculating candidate 3-D coordinates which areconsidered to correspond to the feature point coordinates of a featurepoint in the main image, using the feature point coordinates in the mainimage, the feature point coordinates in the subsidiary images, and theparameters obtained by the parameter calculation means; a 3-Dcoordinates determination means for determining genuine 3-D coordinateswhich correspond to the feature point in the main image, using thecandidate 3-D coordinates calculated by the 3-D coordinates calculationsections in the 3-D coordinates calculation means; a judgment means forjudging whether or not the genuine 3-D coordinates corresponding to thefeature point in the main image is in a designated 3-D space domain, andaccepting the feature point if the genuine 3-D coordinates correspondingto the feature point is judged to be in the designated 3-D space domain;an outline point extraction means for extracting subject outline pointsfrom a plurality of feature points which have been accepted by thejudgment means; and a subject extraction means for extracting part ofthe main image which is surrounded by the subject outline points as thesubject image.
 2. A subject image extraction device as claimed in claim1, wherein the feature point extraction sections in the feature pointextraction means extract edge-like parts in the images where suddenchanges occur in color intensity or luminance as the feature pointcoordinates, in the extraction of the feature point coordinates from themain image or the subsidiary images.
 3. A subject image extractiondevice as claimed in claim 1, wherein the parameter calculation meanscalculates eleven or more parameters per one image pickup section.
 4. Asubject image extraction device as claimed in claim 1, wherein the 3-Dcoordinates determination means determines the genuine 3-D coordinatesby selecting a group of 3-D coordinates from each of the candidate 3-Dcoordinates calculated by each of the 3-D coordinates calculationsections so that the sum of distances between the 3-D coordinatesincluded in the group will become the smallest and defining the genuine3-D coordinates by the average of the 3-D coordinates included in thegroup.
 5. A subject image extraction device as claimed in claim 1,further comprising an image pickup means including the main image pickupsection and the two or more subsidiary image pickup sections.
 6. Asubject image extraction device as claimed in claim 1, furthercomprising a subject display means for displaying the part of the mainimage which has been extracted by the subject extraction means.
 7. Asubject image extraction device as claimed in claim 1, wherein thenumber of the image pickup sections is three.
 8. A subject imageextraction device comprising: an image storage means including mainimage storage section for storing images which have been picked up by amain image pickup section for picking up a main image from which animage of a subject is extracted, and two or more subsidiary imagestorage sections for storing images which have been picked up by two ormore subsidiary image pickup sections for picking up subsidiary imagesto be referred to in subject image extraction; to a feature pointneighborhood extraction means including three or more feature pointneighborhood extraction sections for extracting feature pointcoordinates in the images which have been stored in each of the imagestorage sections and extracting color/luminance information of pixels inthe neighborhood of the feature point coordinates; a parametercalculation means for calculating parameters which represent relativepositions and attitudes of the main image pickup section and thesubsidiary image pickup sections, using the feature point coordinates inthe images of a 3-D object on which a predetermined pattern is drawnwhich have been picked up by the main image pickup section and thesubsidiary image pickup sections; a corresponding point determinationmeans including two or more corresponding point determination sectionsfor selecting a group of candidate corresponding points in thesubsidiary image which are considered to correspond to the feature pointin the main image, from the feature points in the subsidiary images,using the feature point coordinates in the main image and the subsidiaryimage and the color/luminance information of pixels in the neighborhoodof the feature point coordinates which have been extracted by thefeature point neighborhood extraction means; a 3-D coordinatescalculation means including two or more 3-D coordinates calculationsections for calculating candidate 3-D coordinates which are consideredto correspond to a feature point in the main image, using the featurepoint coordinates in the main image, the group of candidatecorresponding points, and the parameters obtained by the parametercalculation means; a 3-D coordinates determination means for determininggenuine 3-D coordinates which correspond to the feature point in themain image, using the candidate 3-D coordinates calculated by the 3-Dcoordinates calculation sections in the 3-D coordinates calculationmeans; a judgment means for judging whether or not the genuine 3-Dcoordinates corresponding to the feature point in the main image is in adesignated 3-D space domain, and accepting the feature point if thegenuine 3-D coordinates corresponding to the feature point is judged tobe in the designated 3-D space domain; an outline point extraction meansfor extracting subject outline points from a plurality of feature pointswhich have been accepted by the judgment means; and a subject extractionmeans for extracting part of the main image which is surrounded by thesubject outline points.
 9. A subject image extraction device as claimedin claim 8, wherein the feature point neighborhood extraction sectionsin the feature point extraction means extract edge-like parts in theimages where sudden changes occur in color intensity or luminance as thefeature point coordinates, in the extraction of the feature pointcoordinates from the main image or the subsidiary images.
 10. A subjectimage extraction device as claimed in claim 8, wherein the parametercalculation means calculates eleven or more parameters per one imagepickup section.
 11. A subject image extraction device as claimed inclaim 8, wherein the 3-D coordinates determination means determines thegenuine 3-D coordinates by selecting a group of 3-D coordinates fromeach of the candidate 3-D coordinates calculated by each of the 3-Dcoordinates calculation sections so that the sum of distances betweenthe 3-D coordinates included in the group will become the smallest anddefining the genuine 3-D coordinates by the average of the 3-Dcoordinates included in the group.
 12. A subject image extraction deviceas claimed in claim 8, further comprising an image pickup meansincluding the main image pickup section and the two or more subsidiaryimage pickup sections.
 13. A subject image extraction device as claimedin claim 8, further comprising a subject display means for displayingthe part of the main image which has been extracted by the subjectextraction means.
 14. A subject image extraction device as claimed inclaim 8, wherein the number of the image pickup sections is three.